Audio-Visual Tracking of Concurrent Speakers

Audio-visual tracking of an unknown number concurrent speakers in 3D is a challenging task, especially when sound and video are collected with compact sensing platform. In this paper, we propose tracker that builds on generative discriminative audio-visual likelihood models formulated particle filtering framework. We localize multiple de-emphasized acoustic map assisted by the image detection-derived observations. The multi-modal observations either assigned to existing tracks for computation or used initialize new tracks. likelihoods rely color distribution target value. Experiments AV16.3 CAV3D datasets show proposed outperforms uni-modal trackers state-of-the-art approaches both plane.